oracle.hadoop.loader.examples

## Class CSVInputFormat

- java.lang.Object
- - org.apache.hadoop.mapreduce.InputFormat<java.lang.Object,org.apache.avro.generic.IndexedRecord>
  - - oracle.hadoop.loader.examples.CSVInputFormat

-     public class CSVInputFormat
      extends org.apache.hadoop.mapreduce.InputFormat<java.lang.Object,org.apache.avro.generic.IndexedRecord>

  This is a simple `InputFormat` example for CSV files. It uses `TextInputFormat` to break the input file(s) into lines, then breaks each line into fields using a comma (,) separator, and places the fields into an Avro `IndexedRecord`.

  CSVInputFormat uses the following simple Avro schema:

       {
          "type" : "record",
          "name" : "All_string_schema",
          "fields" : [
              {"name" : "F0", "type" : "string"},
              {"name" : "F1", "type" : "string"},
              ...]
       }

  Note that:

  - the schema is of type record;
  - all fields are of type string;
    and
  - field names are F0, F1, F2, etc. A field "Fi" will be loaded only if:
    - the loaderMap file specifies a COLUMN element with the attribute: field="Fi"
      or
    - the target table has a column named Fi (and the 'oracle.hadoop.loader.targetTable' property was used in lieu of a loaderMap file).

  CSVInputFormat cannot extend TextInputFormat, since `TextInputFormat.createRecordReader()` returns RecordReader <LongWritable,Text>, while `createRecordReader(org.apache.hadoop.mapreduce.InputSplit, org.apache.hadoop.mapreduce.TaskAttemptContext)` has to return RecordReader <Object,IndexedRecord>.

  The `Object` key is used only when there is an error processing the corresponding `IndexedRecord` value. The key's `toString()` method is used to generate a printable message helping you identify the culprit record. If the key is null, then no information identifying the record will be printed if the record fails.

  You can use the key to print:

  - data file URI,
  - InputSplit information,
  - data file and the record's offset in that file,
  - textual representation of the actual record (not advisable if data contains sensitive information, as it may be printed in Hadoop logs throughout the cluster).
  - etc.

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `static class` | `CSVInputFormat.CSVRecordReader` The record reader parses the input data into key/value pairs which are read by `OraLoaderMapper`.

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `CSVInputFormat()`

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `org.apache.hadoop.mapreduce.RecordReader<java.lang.Object,org.apache.avro.generic.IndexedRecord>` | `createRecordReader(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context)` Create a record reader for a given split.
    `static org.apache.avro.Schema` | `generateSimpleAllStringSchema(int numFields)` Generate an Avro Record schema for the CSV input record.
    `java.util.List<org.apache.hadoop.mapreduce.InputSplit>` | `getSplits(org.apache.hadoop.mapreduce.JobContext context)` Logically split the set of input files for the job.

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### CSVInputFormat

          public CSVInputFormat()

  - ### Method Detail

    - #### getSplits

          public java.util.List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext context)
                                                                           throws java.io.IOException

      Logically split the set of input files for the job.
      Specified by:
        `getSplits` in class `org.apache.hadoop.mapreduce.InputFormat<java.lang.Object,org.apache.avro.generic.IndexedRecord>`
      Parameters:
        `context` - job configuration.
      Returns:
        an array of `InputSplit`s for the job.
      Throws:
        `java.io.IOException`

    - #### createRecordReader

          public org.apache.hadoop.mapreduce.RecordReader<java.lang.Object,org.apache.avro.generic.IndexedRecord> createRecordReader(org.apache.hadoop.mapreduce.InputSplit split,
                                                                                                                            org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                                                                              throws java.io.IOException,
                                                                                                                                     java.lang.InterruptedException

      Create a record reader for a given split. The framework will call `RecordReader.initialize(InputSplit, TaskAttemptContext)` before the split is used.
      Specified by:
        `createRecordReader` in class `org.apache.hadoop.mapreduce.InputFormat<java.lang.Object,org.apache.avro.generic.IndexedRecord>`
      Parameters:
        `split` - the split to be read
        `context` - the information about the task
      Returns:
        a new record reader
      Throws:
        `java.io.IOException`
        `java.lang.InterruptedException`

    - #### generateSimpleAllStringSchema

          public static org.apache.avro.Schema generateSimpleAllStringSchema(int numFields)

      Generate an Avro Record schema for the CSV input record.
      Parameters:
        `numFields` - the number of fields for the schema
      Returns:
        a simple Avro schema:

             {
                "type" : "record",
                "name" : "All_string_schema",
                "fields" : [
                    {"name" : "F0", "type" : "string"},
                    {"name" : "F1", "type" : "string"},
                    ...]
             }

Copyright © 2011, 2015, Oracle and/or its affiliates. All rights reserved.